Bayesian Statistics

In the absence of any evidence, the credence—the degree of belief—should be equally distributed among all possible outcomes.

• The certainty upon the system. Strong prior would change little.

• \[ \operatorname{P}[A|B] = \frac{\operatorname{P}[B|A]\operatorname{P}[A]}{\operatorname{P}[B]} \] where \(\operatorname{P}\) is the probability of the events \(A\) and \(B\).
• According to the Bayesian probability interpretation:
  • \(\operatorname{P}[A|B]\) is the posterior probability of \(A\) given \(B\).
  • \(\operatorname{P}[B|A]\) is the likelihood of \(A\) given a fixed \(B\), since \(\operatorname{P}[B|A] = \operatorname{L}[A|B]\).
  • \(\operatorname{P}[A]\) is the prior probability.
  • \(\operatorname{P}[B]\) is the marginal probability.

The maximum likelihood estimate of \(\theta\): \[ \hat{\theta}_{\rm MLE}(x) = \operatorname*{arg\ max}_{\theta} f(x\mid\theta) \] can be generalized to include the prior distribution \(g(\theta)\) using Bayes' theorem:

\begin{align*} \hat{\theta}_{\rm MAP}(x) &= \operatorname*{arg\ max}_{\theta}\frac{f(x\mid \theta)g(\theta)}{\int_\Theta f(x\mid\vartheta)g(\vartheta)d\vartheta} \\[10pt] &= \operatorname*{arg\ max}_{\theta} f(x\mid \theta)g(\theta). \end{align*}